Skip to content

3.1.0 Released#326

Closed
yogacc33 wants to merge 88 commits intomainfrom
dev
Closed

3.1.0 Released#326
yogacc33 wants to merge 88 commits intomainfrom
dev

Conversation

@yogacc33
Copy link
Copy Markdown
Contributor

@yogacc33 yogacc33 commented Mar 20, 2025

Modification

3.1.0 Released.

yogacc33 and others added 30 commits February 21, 2025 22:33
@yogacc33 @e06084
Framework simplified v1.0 (#211)
61fbdce 
Co-authored-by: chupei <e06084@126.com>
@ideaflow
add zh-en-article quality model (#208)
29445fa 
Co-authored-by: ningwenchang <ningwenchang@pjlab.org.cn>
@darkrush
bugfix: Make CleanExp inherit from WebKitBaseException (#223)
fd04da5
@feifei2023
fix html simplify noscript tag (#222)
83fa499 
Co-authored-by: jinzhenj <jinzhenjiang@pjlab.org.cn>
@e06084
[CI]: add python3.12&3.11 env (#220)
917c466
@dt-yy
resolve nest table (#225)
cb8b5fb 
Co-authored-by: quyuan <quyuan@pjlab.org>
@lsp213
fix: mml to latex and math extract no full (#219)
4765cfb
@drunkpig
fix: DataJson construct method do not change outer variable (#226)
9892563 
Co-authored-by: drunkpig <xuchao@pjlab.org.cn>
@NgZiming
fix: code test case (#221)
74f4383 
Co-authored-by: wuziming <wuziming@sensetime.com>
@e06084
[fix]: fix math nonetype (#229)
c499c58
@ideaflow
add document of quality model (#230)
0ae2776 
Co-authored-by: ningwenchang <ningwenchang@pjlab.org.cn>
@lsp213
fix:math no text (#233)
002f9de
@darkrush
feat: add CleanModule to provide general clean function (#234)
4e63cc5
@ideaflow @darkrush
Add a new interface in political model to accommodate the new require…
6d48387 
…ment (#210)

Co-authored-by: ningwenchang <ningwenchang@pjlab.org.cn>
Co-authored-by: darkrush <qiujiantao@hotmail.com>
@e06084
feat: add code and math test html in st (#235)
211759e
@e06084
[CI]: python3.11 and 3.12 run when requirements modify (#236)
47bad53
@ideaflow
Revise quality document，and change english stop words reading method (#…
840c7bd 
…241)

Co-authored-by: ningwenchang <ningwenchang@pjlab.org.cn>
@darkrush
feat: add CleanModule to provide general clean function (#246)
6002dfc
@2471023025
lang_id doc revise (#245)
3f4b8e8 
Co-authored-by: huyc <huyucheng1@pjlab.org.cn>
@darkrush
feat: add CleanModule to provide general clean function (#232)
b095987
@darkrush
fix: move html_simplify_classify.md to docs/llm_web_kit/model (#247)
90cc8a2
@yogacc33
Exception refactoring (#249)
44a105e
@darkrush
refactor: remove old clean exception add model-related exceptions (#252)
1e97c9e
@darkrush
feat: 添加 CleanModule 文档及参数说明，支持内容质量清洗 (#250)
be90fe7
@2471023025
修改语言分类模型文档 (#251)
da6a8a0 
Co-authored-by: huyc <huyucheng1@pjlab.org.cn>
@Adela-Yu-Coder
init unsafe_words_detector.py (#194)
68537e3 
Co-authored-by: yujing <yujing@pjlab.org.cn>
@darkrush
feat: 添加线程安全的文件下载功能，支持文件哈希校验和锁机制 (#254)
68a18be
@darkrush
fix: 修复文件锁定机制，确保锁文件在异常情况下被正确删除 (#256)
fd8e2a0
@2471023025
Dev lid218 (#255)
97c15b2 
Co-authored-by: huyc <huyucheng1@pjlab.org.cn>
@dt-yy
调整解析顺序&更新标准 (#248)
340f11c 
Co-authored-by: quyuan <quyuan@pjlab.org>
darkrush and others added 29 commits March 12, 2025 12:37
@darkrush
Merge pull request #296 from darkrush/basemodel
cce1eee 
mock CACHE_TMP_DIR in test cases
@LollipopsAndWine
feat: 修复获取文本未保留换行问题、增加段落文本的测试用例 (#297)
f758d35
@2471023025
feat: use the first item in predict result as langurage_details (#298)
77b5e5e 
Co-authored-by: huyc <huyucheng1@pjlab.org.cn>
@darkrush
fix: 修复空内容返回的语言详情，在176版本下language_details返回"not_defined" (#300)
72f5037
@darkrush
fix: set mock CACHE_TMP_DIR to /tmp correctly
18286da
@e06084
fix: empty extract fomula (#304)
c1b2fda
@e06084
[fix]: fix 一些特殊错误的公式 (#299)
32ed991
feat: page classify
dcdb988
feat: classify page by html layout use GPU
3f210f6
feat: raise CleanModelUnsupportedLanguageException in clean module
efda13e
fix: clean model exception
4fdad12
@drunkpig
feat: page classify
4702438 
feat: page classify
fix: code and text unit test
4dbcecf
fix: pipeline config move from source code to file
3ee2fba
@drunkpig
Merge pull request #310 from drunkpig/dev9
7f4fc08 
fix: code and text unit test
@e06084
feat: math extract support mjx-container tag (#311)
5039964
@shijinpjlab
feat: add MM_NODE_LIST in to_nlp_md (#305)
cbcb1fc
@drunkpig
feat: simple user api to extract html to markdown (#313)
8d0f7f3 
Co-authored-by: drunkpig <xuchao@pjlab.org.cn>
@drunkpig
feat: add title to DataJson (#314)
caf7159 
Co-authored-by: drunkpig <xuchao@pjlab.org.cn>
@e06084
fix: html parser support xml_declaration (#315)
3349d50
@e06084
[feat] support math extract from mathjax config (#303)
9200551
@Adela-Yu-Coder @darkrush
Update rule-based safety and model-based safety
53d4013 
Co-authored-by: yujing <yujing@pjlab.org.cn>
Co-authored-by: qiujiantao <qiujiantao@hotmail.com>
@LollipopsAndWine
feat: 将换行更换为双换行、配置全局常量 (#320)
c46934c
@yogacc33
Revert "feat: 将换行更换为双换行、配置全局常量 (#320)" (#321)
1f9c2ca
@yogacc33
build: setup add config files (#319)
584a355
@dt-yy @NgZiming
性能提升&问题修复 (#317)
1b18519 
Co-authored-by: quyuan <quyuan@pjlab.org>
Co-authored-by: 吴梓铭 <ngziming2014@outlook.com>
Co-authored-by: wuziming <wuziming@sensetime.com>
@e06084
bench: update main_html extractor (#318)
d360ff3
@dt-yy @NgZiming
update version (#323)
4a3928b 
Co-authored-by: quyuan <quyuan@pjlab.org>
Co-authored-by: 吴梓铭 <ngziming2014@outlook.com>
Co-authored-by: wuziming <wuziming@sensetime.com>
@LollipopsAndWine
feat: 将获取段落文本时，br换成双换行 (#325)
ce41728
@yogacc33 yogacc33 closed this Mar 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.